w NGA Center for 

f BEST PRACTICES 

V 




ssue Brief 





Contact: Dane Linn 
Education Division 
202/624-3629 
November 9, 2009 



Measuring Student Achievement at Postsecondary Institutions 

Executive Summary 

Existing measures of student achievement exclude far too many students and track too few 
student milestones. These exclusions can penalize institutions for serving disadvantaged students 
by understating these students’ performance. 

This Issue Brief offers state leaders a formula for measuring systemwide student achievement at 
postsecondary institutions. Although many areas of postsecondary performance must be 
measured, student learning is core to the mission of the postsecondary system and instructional 
costs are the single largest expense for public postsecondary institutions. 1 For these reasons, this 
Issue Brief focuses on measuring student achievement as a crucial indicator of postsecondary 
performance. 

Governors can take four steps to design a postsecondary education data system that meets the 
needs of their state: 

Step 1 : Select appropriate student milestones to measure remediation, retention, and 
attainment. 

Step 2: Determine which students to count. 

Step 3 : Select appropriate benchmarks. 

Step 4: Group achievement rates by student population and institution. 

The four key student achievement milestones all states shoidd track are: 

• Successful completion of remedial and core courses; 

• Advancement from remedial to credit-bearing courses; 

• Transfer from a two-year institution to a four-year institution; and 

• Credential attainment. 

Improving the performance of postsecondary institutions, including community colleges, has 
increasingly taken center stage in state economic recovery plans. Nationwide, governors, state 
higher education officers, foundations, industry leaders, and education advocacy groups are 
partnering with President Barack Obama and the U.S. Department of Education to ensure the 
economy has the highest percentage of college-educated workers in the world. 2 

Achieving this goal will require easily accessible postsecondary data systems with well-defined 
performance measures, yet many state leaders have inadequate or misleading data on the 
performance of their postsecondary institutions. With 4,352 degree-granting postsecondary 
institutions in the United States — including private, public, two-year, and four-year institutions — 
reliably measuring postsecondary performance is no easy task. 3 
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Why Current Measurements Are Inadequate 

Although many states continue to improve their postsecondary data systems, uneven data quality 
and imprecise performance measures have stalled the improvement of postsecondary education. 
Governors need reliable measures to allocate funds appropriately, measure the progress of 
postsecondary performance over time, encourage successful practices and discourage waste in the 
postsecondary system, and protect the future of the state economy by maintaining a healthy 
postsecondary system. 

Current measurements of postsecondary student achievement are inadequate because: 

• Federal data exclude too many students; 

• Many important student milestones are not measured; and 

• Data are not disaggregated enough to reveal appropriate points of intervention. 

Federal Data Exclude Too Many Students 

The most commonly cited statistics on student completion are collected through the Student Right 
to Know and Campus Security Act of 1990 (SRK). The SRK graduation rate is calculated by 
counting all first-time, full-time students who earn a degree or certificate at the college they first 
enrolled at within 150 percent of the expected completion time and dividing that count by the 
total cohort with which the graduates first enrolled. 

This outdated and narrow federal graduation rate excludes nearly half of community college and 
public university students, because it does not count part-time students and students who transfer 
among community colleges. According to the Community College Research Center, up to 40 
percent of community college students enroll in multiple institutions during the six years they are 
tracked, but students who transfer will often not be counted even if they complete their degree at 
a new institution. 4 Moreover, more than one-third of all community college students also are part- 
time enrollees, so SRK data is an incomplete measure of student achievement. 

Many Important Student Milestones Are Not Measured 

The most commonly used measure for public higher education funding formulas is total student 
enrollment. This measure creates no incentive to see students through to completion — year to 
year, semester to semester, or even to the end of a single course. Attempted credits, rather than 
completed credits, are all that matter for state funding so, not surprisingly, little investment is 
made in improving student completion rates. 

Alternatively, strict graduation rate formulas can penalize schools that serve disadvantaged 
students because these schools will inevitably have lower graduation rates. Moreover, a singular 
emphasis on graduation can discourage open-enrollment policies, because skimming top students 
will improve institutional performance despite excluding students who may benefit most from 
postsecondary education. 5 Graduation rate funding formulas may also pressure schools to lower 
their graduation standards if they are desperate for funds and are not meeting graduation targets. 

Clearly, neither enrollment totals nor graduation rates are adequate as sole indicators of 
institutional performance. Intermediate student milestones that have consistently been linked to 
student completion are necessary for determining at what points in the education pipeline 
postsecondary students are dropping out and where education policymakers could intervene. 

Data Are Not Disaggregated Enough to Reveal Appropriate Points of Intervention 

Many states also lack the ability to compare student subpopulations or institutions. 6 The absence 
of cross-institutional data systems with student-level information and compulsory reporting 
requirements prevents governors from identifying failure or success at the postsecondary level. 7 
Many states cannot compare individual institutions within a statewide system, or make 
comparisons across public and private institutions. Comparing the performance of at-risk student 
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populations (e.g., students with limited English proficiency) with state averages also is important 
for policymakers, but too often these subpopulations are not tracked by existing data systems. 

How States Can Improve Their Measurements 

The following four steps offer state leaders a template for developing postsecondary student 
achievement measures (see, also, the appendix). Because many states are at different stages in 
adopting postsecondary data systems, each step is broken down into key measures and more 
advanced measures for states with greater capacity. States also have distinct privacy laws that 
may limit the type of data that can be collected, distinct governance structures for decision 
making, and distinct financial constraints; this template was designed with such broad differences 
in mind. 

Step 1: Select appropriate student milestones. 

Step 2: Determine which students to count. 

Step 3 : Select appropriate benchmarks. 

Step 4: Group achievement rates by student population and institution. 

Step 1: Select Appropriate Student Milestones 

The National Governors Association Center for Best Practices (NGA Center) recommends that all 
states track four key student achievement milestones. 8 The NGA Center has selected these 
milestones based on their proven relationship to student persistence and success. Governors can 
then use the milestones to target resources to the most effective institutions. Student achievement 
milestones also enable postsecondary institutions to explain their own productivity gains to state 
leaders and add credibility to their requests for funding. 

The four key student achievement milestones all states should track are: 

• Successful completion of remedial and core courses; 

• Advancement from remedial to credit-bearing courses; 

• Transfer from a two-year to a four-year institution; and 

• Credential attainment. 

Remediation Milestones. Remediation milestones measure the incremental achievements of 
students en route to degree attainment, focusing on the advancement of students who begin at 
community colleges unable to complete college-level work. This characterization applies to a 
majority of community college students — 59 percent of them require at least one remedial course 
during their first year. 9 Completion of remedial courses is crucial for credential attainment, 
because remedial courses, often called developmental courses or basic skills courses, do not count 
toward an associate degree or a bachelor’s degree. By measuring student remediation, states also 
assess the ability of their K-12 system to prepare students for college-level work. 

The NGA Center’s recommended remediation milestones are selected based on their association 
with future success and degree completion; for example, students who move from part-time to 
full-time enrollment are more likely to complete their degree programs. 10 This makes 
advancement from part-time to full-time status a valuable remediation milestone that can be 
incentivized through state funding formulas. In Tennessee, the Higher Education Commission 
tracks the percentage of students who enroll in developmental courses and successfully complete 
college credit-bearing coursework the following year. The Wyoming Community College 
Commission tracks the percentage of students who enroll in remedial courses and the percentage 
of students who pass remedial courses. 11 Wyoming is then able to determine whether its K-12 
system is adequately preparing high school graduates for college-level work. 

The key remediation milestones are pass rates in remedial and core courses and advancement 
from remedial to credit-bearing courses. A more developed measurement system will also track 
advancement from English language learner courses to mainstream courses. 
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Retention Milestones. Retention milestones track student persistence, a prerequisite for student 
success. Persistence can be demonstrated through enrollment status — part time or full time — as 
well as sustained enrollment over multiple semesters rather than intermittent enrollment. 
Enrollment status and semester-to-semester enrollment are both crucial indicators of time to 
degree. Moreover, the rise in national time-to-degree averages dramatically increases the cost of 
providing each degree, because the state must fund college or university overhead for five or six 
years, rather than four years, per bachelor’s degree. 

The Ohio Board of Regents reports student retention not only at the institution level, but also 
across the state higher education system, so students who transfer from their original institution 
are not dropped from the database. Texas tracks students who leave school but return within one 
to two years, as does North Carolina. Setting student census reporting deadlines at the end, 
rather than at the beginning, of the academic year also creates strong incentives for institutions to 
improve course completion rates. 12 

The key retention milestone is transfer from a two-year institution to a four-year institution. A 
more developed measurement system will also track semester-to-semester continuous enrollment, 
year-to-year continuous enrollment, and advancement from part-time to full-time status. 

Attainment Milestones. Attainment milestones measure degree completion and certificate 
completion. These milestones are the most commonly used markers of student achievement. 
Although credential attainment should not be the only measure of student achievement, it is 
nonetheless the ultimate goal for most postsecondary students. Both students and states benefit 
from degree completion, because enrollment without completion has negligible effects on 
earnings and, therefore, tax revenues. 13 States with higher percentages of adults with 
postsecondary credentials also show higher rates of workforce participation, higher rates of 
volunteerism, superior health outcomes, and reduced participation in Medicaid. 14 

Current measures of credential attainment can be improved by adopting longitudinal data systems 
that link degree or certificate attainment with future employment and changes in earnings. 
Longitudinal data would also enable cohort studies that follow students over time. Estimated 
snapshot graduation rates simply divide the current year’s graduates by the size of the freshman 
class four years prior. 

The key attainment milestones are attainment of a bachelor’s degree, attainment of an associate 
degree, and attainment of a credential below an associate degree. A more developed measurement 
system will also track obtainment of a job in the field of study or obtainment of a job that requires 
the most recently attained credential. 

Step 2: Determine Which Students to Count 

Many long-held stereotypes about college students — that they live on campus, enroll full-time, 
and graduate in four years — fail to describe the U.S. college student population and exclude most 
community college students. Understanding the changing demographics of postsecondary 
students is necessary for choosing appropriate student achievement milestones and benchmarks. 
Meaningful data on postsecondary students should track part-time students, full-time students, 
first-time students, transfer students, and students pursuing education for non-credential 
purposes. 15 

Thirty-one percent of all part-time and full-time students seeking an associate degree or a 
bachelor’s degree are above age 24 (see Figure 1). These older students are less likely to be 
enrolled full time (see Figure 2). 
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Figure 2: Students Enrolled in 
Postsecondary Institutions by 
Enrollment Status 
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Source: U.S. Department of Education, National Center for Education Statistics, Integrated Postsecondary Education 
Data System, spring 2007. 





Although part-time enrollment makes it nearly impossible to complete a bachelor’s degree in four 
years, 37 percent of all students seeking an associate degree or a bachelor’s degree are enrolled 
part time. 16 

Nearly one out of every two students enrolled at a public postsecondary institution is in a two- 
year college, but both state and federal data are misleading when applied to two-year colleges. 17 
Students at two-year colleges often transfer among community colleges or take time off between 
community college and transferring to a four-year institution, so reliable data on institutional 
performance shoidd not drop these students. 

Step 3: Select Appropriate Benchmarks for Each Milestone 

Each remediation, retention, and attainment milestone should be benchmarked against an agreed- 
upon standard of achievement. Currently, the most common benchmark used to assess student 
achievement is a 100 percent student graduation rate within four years; little or no attention is 
focused on identifying or benchmarking remediation and retention milestones. Successful 
benchmarks enable state policymakers to compare student achievement at in-state postsecondary 
institutions with student achievement at peer institutions outside the state, with student 
achievement averages in other states, and with national student achievement averages. 

Michigan uses a variety of criteria, such as SAT score and percentage of Pell recipients, to select 
15 peer institutions outside the state and then compares its postsecondary achievement rates with 
those of the peer institutions. Nebraska compares the performance of its public postsecondary 
institutions with that of 10 peer-level institutions. Massachusetts compares institutional 
performance in two contexts: peer institutions across the United States and national averages. 

An institution’s prior performance can also be a valuable benchmark for rewarding institutional 
improvement across diverse institutions. To increase the total number of bachelor’s degrees 
awarded each year, Indiana provided additional incentive dollars for institutions that surpassed 
their prior year’s total number of graduates. 18 In this instance, Indiana used the prior year’s 
performance as the benchmark and then rewarded institutions an additional fixed amount of 
funding for each additional graduate. 
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Two benchmarks should be used when measuring student achievement: the performance of peer 
institutions, as determined by enrollment, average SAT score, faculty size, and percentage of 
student body eligible for Pell grants; and the institution’s prior-year performance. 

Step 4: Group Achievement Rates by Student Population and Institution 

Aggregate student performance indicators — such as the average graduation rate in a state — often 
conceal the low academic performance of disadvantaged subpopulations. State-level data that 
cannot be broken down by student subpopulation or institution also prevent state leaders from 
identifying appropriate points of intervention. High-quality performance measures should reflect 
the diversity of the postsecondary student population by disaggregating data into a variety of 
subpopulations. 

In Kentucky, disaggregation enables state leaders to see the achievement rates by gender, 
ethnicity, transfer status, and credits earned at each public institution, including two-year colleges 
and technical colleges. Washington is one of many states that disaggregate student achievement 
by Pell eligibility, a useful indicator of student socioeconomic status. These breakdowns of 
student achievement enable state leaders to identify problem areas within the postsecondary 
system and develop policy options to address these deficits. 

States should disaggregate attainment measures by: 

• Part-time and full-time students; 

• Transfer students; 

• Students beginning in remedial courses; 

• Students with limited English proficiency; 

• Pell-eligible students; 

• Underrepresented ethnic minorities; 

• Students in science, technology, engineering, and mathematics (STEM) fields; 

• Students above age 2 1 when first enrolled; and 

• Students with disabilities. 

What Obstacles Remain to Improved Measurement 

Introducing a new system of postsecondary student achievement measures can pose several 
implementation challenges, including institutional resistance, the diverse array of institutions, 
manipulation of data, and legal obstacles. 

Institutional Resistance 

Many postsecondary institutions, both public and private, are accustomed to more autonomy than 
is afforded the K-12 system and will be reluctant to change this arrangement. Moreover, 
declining state funding for public higher education has further deteriorated governors’ influence 
over higher education. Although states remain the largest single contributor to public higher 
education, tuition payments continue to replace state funding — a trend that is expected to 
continue. 19 At public universities, only 24 percent of all revenue comes from state government, 
considerably less than state funding for public K-12 education. 20 

The Diverse Array of Institutions 

A mixed market of public and private institutions — including for-profit and nonprofit 
organizations as well as two-year and four-year institutions — weakens the value of using only one 
measure of student achievement. Many of these institutions have separate goals, ranging from 
large universities that prioritize faculty research and graduate education to technical and 
vocational colleges that offer accelerated certification programs. Despite the appeal of using a 
single measure, such as the federal Student Right to Know graduation rate, comparing such 
diverse institutions requires different measures. 
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Manipulation of Data 

Graduation rates can easily be inflated or manipulated and often require diligent state oversight. 21 
Small technical decisions, such as not counting part-time students, can have dramatic effects on 
graduation rates. Many public offices do not have the capacity to oversee large postsecondary 
student data systems. Moreover, many state staff lack the expertise to interpret postsecondary 
student achievement data. Some states may require third-party providers to meet these needs. 

Legal Obstacles 

Varying state privacy laws and local interpretations of the Family Educational Rights and Privacy 
Act and other federal privacy protections can create red tape for state collection of student-level 
data. Each state will need to consult its legal advisors before pursuing an ambitious data 
campaign that links student-level data across institutions or ties student-level data to workforce 
data. Although tracking individual students over time is enormously valuable for informing state 
policy choices, creating unique student identifiers to facilitate analysis can be a contentious legal 
issue. 

How to Overcome Implementation Challenges 

Improved measurement of postsecondary student achievement is necessary for identifying the 
successes and failures of the postsecondary system. Measurement will more likely to be embraced 
if these principles and strategies are followed: 

• Emphasize shared goals instead of accountability; 

• Use different milestones and benchmarks for two- and four-year schools; 

• Make data more accessible; and 

• Link data systems. 

Emphasize Shared Goals Instead of Accountability 

State policymakers should emphasize shared goals with the leadership of postsecondary 
institutions — such as high performance, improved student outcomes, and increased institutional 
capacity — to establish more willing collaboration. Confrontational messages of accountability 
and state intervention will be met with resistance from postsecondary leadership. The language of 
reform used in the K-12 system cannot easily be extended to the postsecondary system given the 
two systems’ radically different funding and governance structures. 

Use Different Milestones and Benchmarks for Two- and Four-Year Schools 

Separate milestones and benchmarks for two-year and four-year institutions make institutional 
data more meaningful. 22 This Issue Brief has detailed many of the ways that states can customize 
student achievement measures by selecting appropriate remediation, retention, and attainment 
milestones. Two-year institutions should prioritize remediation milestones, while four-year 
institutions should prioritize attainment milestones. States can also use different benchmarks for 
different types of institutions. One approach is to benchmark against an institution’s prior-year 
performance so each institution is competing against itself; this can help prevent unrealistic 
comparisons between elite four-year institutions and open-enrollment colleges. 

Make Data More Accessible 

Student achievement data should be publicly accessible so students and their families can make 
informed choices about the institutions in which students enroll. Currently, most students and 
their families are unaware of the remediation, retention, and graduation rates of the institutions in 
which students are interested in enrolling — information that would likely affect their choice of 
school. By enabling students and their families to know such performance measures, institutions 
will be pressured to improve their student achievement outcomes in an effort to attract students. 
Making data more accessible is a market-driven solution to improving the performance of 
postsecondary institutions. 



Page 8 — Measuring Student Achievement at Postsecondary Institutions 



Link Data Systems 

Many of the current gaps in postsecondary student achievement data result from a diffuse 
reporting system that fails to follow individual students across institutions over time. 
Consolidation would streamline the development of longitudinal student-level data systems and 
could also increase access to student achievement data. 23 Consolidating postsecondary data 
systems with K-12 data systems can also reduce the administrative costs of storing and reporting 
data. 

Conclusion 

Existing measures of student achievement exclude far too many students and track too few 
student milestones. Because current measures were originally designed for four-year institutions, 
two-year institutions are often penalized and have underreported successes. Governors need 
reliable measures to allocate funds appropriately, measure the progress of postsecondary 
performance over time, encourage successful practices, and discourage waste in the 
postsecondary system. 

This Issue Brief provides guidance to state leaders on how to design student achievement 
measures that meet the unique policy priorities of their state. To remedy the lack of accurate data 
on public postsecondary institutions, governors can take these four steps to design a student 
achievement data system: 

Step 1 : Select appropriate student milestones to measure remediation, retention, and 
attainment. 

Step 2 : Determine which students to count. 

Step 3: Select appropriate benchmarks. 

Step 4: Group achievement rates by student population and institution. 

The four key student achievement milestones all states should track are: 

• Successful completion of remedial and core courses; 

• Advancement from remedial to credit-bearing courses; 

• Transfer from a two-year institution to a four-year institution; and 

• Credential attainment. 



This Issue Brief was written by Jeremy Bearer-Friend, a Ph.D. student at the University of 
California, Berkeley, who served as an intern in the NGA Center. 
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Appendix: Four-Step Formula for Measuring Postsecondary Student Achievement 



STEP 1 

SELECT APPROPRIATE 
MILESTONES 


Pass rates in remedial and core courses. 

Advancement from remedial to credit-bearing courses. 
Transfer from a two-year institution to a four-year institution. 
Credential attainment. 


STEP 2 


DETERMINE WHICH 


Part-time students. 


STUDENTS TO COUNT 


Full-time students. 
First-time students. 
Transfer students. 


STEP 3 

SELECT APPROPRIATE 


BENCHMARKS 


Institution’s prior-year performance. 
Peer institutions. 


STEP 4 


GROUP ACHIEVEMENT RATES 


Part-time and full-time students. 


BY STUDENT POPULATION 


Students beginning in remedial courses. 


AND INSTITUTION 


Students with limited English proficiency. 

Pell-eligible students. 

Underrepresented ethnic minorities. 

Students in science, technology, engineering, and mathematics 
(STEM) fields. 

Students above age 2 1 when first enrolled. 

Students with disabilities. 





